Allow generics in runtime expressions by ilevkivskyi · Pull Request #2302 · python/mypy

ilevkivskyi · 2016-10-21T21:01:19Z

As discussed in python/typing#303 generics in typing.py are now cached and are therefore much faster at runtime than before. Here I propose (and it looks like Guido is not against this) to allow generic types as runtime expressions.

@JukkaL Please take a look.

codecov-io · 2016-10-21T21:08:10Z

Current coverage is 83.13% (diff: 100%)

Merging #2302 into master will increase coverage by 0.04%

@@             master      #2302   diff @@
==========================================
  Files            72         72          
  Lines         19032      19036     +4   
  Methods           0          0          
  Messages          0          0          
  Branches       3917       3919     +2   
==========================================
+ Hits          15814      15826    +12   
+ Misses         2618       2611     -7   
+ Partials        600        599     -1

Powered by Codecov. Last update 3ff04c4...1cbd4c5

gvanrossum · 2016-10-24T17:27:48Z

Looks great, no need for Jukka to review.

Fixes python/typing#303. See also python/mypy#2302 (which removes the restriction from mypy). As a motivation, in Python one always can substitute expressions, so that if ``IntNode = Node[int]; IntNode()`` works, then it is reasonable to also allow ``Node[int]``, but say that the first way is preferred.

gvanrossum · 2016-10-24T17:55:32Z

I just got a crash when running this on some code of ours. I'm still trying to distill the code that triggers it into something small, but here's the traceback:

ui/common/xui/wizkit.py:452: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/guido/src/mypy/mypy/__main__.py", line 5, in <module>
    main(None)
  File "/Users/guido/src/mypy/mypy/main.py", line 38, in main
    res = type_check_only(sources, bin_dir, options)
  File "/Users/guido/src/mypy/mypy/main.py", line 79, in type_check_only
    options=options)
  File "/Users/guido/src/mypy/mypy/build.py", line 184, in build
    dispatch(sources, manager)
  File "/Users/guido/src/mypy/mypy/build.py", line 1494, in dispatch
    process_graph(graph, manager)
  File "/Users/guido/src/mypy/mypy/build.py", line 1674, in process_graph
    process_stale_scc(graph, scc)
  File "/Users/guido/src/mypy/mypy/build.py", line 1753, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/guido/src/mypy/mypy/build.py", line 1422, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/Users/guido/src/mypy/mypy/checker.py", line 175, in check_first_pass
    self.accept(d)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 713, in accept
    return visitor.visit_class_def(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 968, in visit_class_def
    self.accept(defn.defs)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 774, in accept
    return visitor.visit_block(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1059, in visit_block
    self.accept(s)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 532, in accept
    return visitor.visit_func_def(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 445, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/Users/guido/src/mypy/mypy/checker.py", line 508, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/guido/src/mypy/mypy/checker.py", line 628, in check_func_def
    self.accept(item.body)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 774, in accept
    return visitor.visit_block(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1059, in visit_block
    self.accept(s)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 875, in accept
    return visitor.visit_return_stmt(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1496, in visit_return_stmt
    self.check_return_stmt(s)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1510, in check_return_stmt
    typ = self.accept(s.expr, return_type)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 1250, in accept
    return visitor.visit_call_expr(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 1940, in visit_call_expr
    return self.expr_checker.visit_call_expr(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 178, in visit_call_expr
    return self.check_call_expr_with_callee_type(callee_type, e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 238, in check_call_expr_with_callee_type
    e.arg_names, callable_node=e.callee)[0]
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 313, in check_call
    self.infer_arg_types_in_context(None, args)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 395, in infer_arg_types_in_context
    arg_type = self.accept(arg, ctx)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1755, in accept
    return self.chk.accept(node, context)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 1297, in accept
    return visitor.visit_index_expr(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 2067, in visit_index_expr
    return self.expr_checker.visit_index_expr(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1271, in visit_index_expr
    result = self.visit_index_expr_helper(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1277, in visit_index_expr_helper
    return self.accept(e.analyzed)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1755, in accept
    return self.chk.accept(node, context)
  File "/Users/guido/src/mypy/mypy/checker.py", line 234, in accept
    typ = node.accept(self)
  File "/Users/guido/src/mypy/mypy/nodes.py", line 1663, in accept
    return visitor.visit_type_application(self)
  File "/Users/guido/src/mypy/mypy/checker.py", line 2079, in visit_type_application
    return self.expr_checker.visit_type_application(e)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 1377, in visit_type_application
    return self.apply_generic_arguments(tp, tapp.types, tapp)
  File "/Users/guido/src/mypy/mypy/checkexpr.py", line 885, in apply_generic_arguments
    return applytype.apply_generic_arguments(callable, types, self.msg, context)
  File "/Users/guido/src/mypy/mypy/applytype.py", line 20, in apply_generic_arguments
    tvars = callable.variables
AttributeError: 'AnyType' object has no attribute 'variables'
ui/common/xui/wizkit.py:452: note: use --pdb to drop into pdb

The line triggering it has:

        # This magical WizkitStrings metaclasss confuses mypy
        return self.account.render_string(WizkitStrings[string], # type: ignore
                flip_role=flip_role, max_len=max_len, **kw)

The class WizkitStrings inherits from another class which has a metaclass defining __getitem__. The metaclass is added using @add_metaclass(...). There may also be an import cycle or a suppressed import (-s) involved.

ilevkivskyi · 2016-10-24T18:07:13Z

@gvanrossum
I think I understand what is going on. The class is not a "real" generic, but WizkitString[string] is parsed as TypeApplication. After #type: ignore it becomes Any. It looks like an easy fix would be to add check if isinstance(tp, AnyType): return AnyType() in visit_type_application. Will try this soon. Should I open a separate PR for this?

gvanrossum · 2016-10-24T20:08:10Z

I figured it was something like that. If you can come up with a test and a fix for that test, a separate PR would be great.

ilevkivskyi · 2016-10-24T20:11:46Z

@gvanrossum Sorry, it is more complex than I thought. I still could not reproduce this crash.
But I believe that #2309 should help (in any case I think it is safer to check for AnyType() or something invalid just to avoid crash and continue checking with AnyType() inferred).
Could you please try that PR with your code?

I will continue to look for a good test for this.

As discussed in python/typing#303 generics in ``typing.py`` are now cached and are therefore much faster at runtime than before. Here I propose (and it looks like Guido is not against this) to allow generic types as runtime expressions.

See #2302 (comment)

Allow generics in runtime expressions

c433cf8

ilevkivskyi mentioned this pull request Oct 21, 2016

Soften restriction for runtime generics in PEP 484 python/peps#120

Merged

ilevkivskyi added 2 commits October 21, 2016 23:47

Add few more tests

baea55d

Re-run Travis

1cbd4c5

gvanrossum merged commit 5b2d3b3 into python:master Oct 24, 2016

JukkaL mentioned this pull request Oct 24, 2016

Document use of generics in expressions #2307

Closed

gvanrossum pushed a commit that referenced this pull request Oct 26, 2016

Cleaner (?) fix for crash in #2302

b9ecdf5

See #2302 (comment)

gvanrossum mentioned this pull request Oct 26, 2016

Cleaner (?) fix for crash in #2302 #2333

Merged

gvanrossum added a commit that referenced this pull request Oct 26, 2016

Cleaner (?) fix for crash in #2302 (#2333)

fc94151

See #2302 (comment)

rwbarton mentioned this pull request Oct 27, 2016

Aliases for generic class applications get type Any #2349

Closed

ddfisher mentioned this pull request Oct 28, 2016

Disallow isinstance of generic expressions #2373

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow generics in runtime expressions#2302

Allow generics in runtime expressions#2302
gvanrossum merged 3 commits intopython:masterfrom
ilevkivskyi:master

ilevkivskyi commented Oct 21, 2016

Uh oh!

codecov-io commented Oct 21, 2016 •

edited

Loading

Uh oh!

gvanrossum commented Oct 24, 2016

Uh oh!

gvanrossum commented Oct 24, 2016

Uh oh!

ilevkivskyi commented Oct 24, 2016

Uh oh!

gvanrossum commented Oct 24, 2016

Uh oh!

ilevkivskyi commented Oct 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ilevkivskyi commented Oct 21, 2016

Uh oh!

codecov-io commented Oct 21, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Current coverage is 83.13% (diff: 100%)

Uh oh!

gvanrossum commented Oct 24, 2016

Uh oh!

gvanrossum commented Oct 24, 2016

Uh oh!

ilevkivskyi commented Oct 24, 2016

Uh oh!

gvanrossum commented Oct 24, 2016

Uh oh!

ilevkivskyi commented Oct 24, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov-io commented Oct 21, 2016 •

edited

Loading